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Algebraic analysis of minimal representations 

Dedicated to Mikio Sato whose pioneering work in 
algebraic analysis has been an inspiration for me. 

By 

Toshiyuki Kobayashi * 



Abstract 



Small representations of a group bring us to large symmetries in a representation 
space. Analysis on minimal representations utilises large symmetries in their geomet- 
ric models, and serves as a driving force in creating new interesting problems that 
interact with other branches of mathematics. 

This article discusses the following three topics that arise from minimal repre- 
sentations of the indefinite orthogonal group: 

1. construction of conservative quantities for ultra-hyperbolic equations, 

2. quantative discrete branching laws, 

3. deformation of the Fourier transform 

with emphasis on the prominent roles of Sato's idea on algebraic analysis. 



§1. Introduction 

The aim of this article is to highlight the prominent roles of Sato's idea on 
hyperfunctions and 2?-modules in the new developments on analysis of minimal 
representations [4, 35, 36, 40]. 
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Minimal representations are the simplest, infinite dimensional 'unipotent 
representations'. They are building blocks of unitary representations. Segal- 
Shale- Weil representation is a classic example for the split simple group of type 
C . There has been an active study on minimal representations of reductive 
groups, mostly by algebraic approaches since 1990s both in the real and in the 
p-adic fields [5, 6, 10, 12, 22, 23, 37, 42, 43, 46, 50, 51, 52]. 

On the other hand, I believe that geometric analysis on minimal represen- 
tations is also a promising area, and have been advocating its study based on 
the following change of viewpoints: 

small representations of a group 
= large symmetries in a representation space. (1-1) 

The terminology 'minimal representations' is defined inside representation the- 
ory (i.e. the annihilator in the universal enveloping algebra is the Joseph ideal, 
see e.g. [10]), and the corresponding 'largest symmetries' are expected to serve 
as a driving force in creating new interesting areas of mathematics even outside 
representation theory. 

The 'largest symmetries' in representation spaces of minimal represen- 
tations may be also observed in branching laws. Indeed, as we shall see in 
Theorem 3.4, it may well happen that broken symmetries of minimal repre- 
sentations reduce to analysis on certain semisimple symmetric spaces (see also 
[30, 39, 41]). This observation indicates that analysis on minimal representation 
involves higher symmetries than those for (traditional) analysis on symmetric 
spaces. 

We focus on the minimal representation of a simple group of type D. This 
is just a single irreducible representation, however, it turns out that geometric 
analysis on its various models is surprisingly rich. Indeed, papers devoted to 
this single representation in very recent years already exceed 500 pages, giving 
rise to the interactions with the following topics: 

• conformal geometry for general pseudo-Riemannian manifolds [30, 38], 

• Dolbeault cohomologies on open complex manifolds [37]. 

• conservative quantities for ultra-hyperbolic equations [40], 

• breaking symmetries and discrete branching laws [39], 

• Schrodinger model and the unitary inversion operator [34, 36], 

• deformation of Fourier transforms [4] , 
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• holomorphic semigroup [33, 35], 

• new special function theory for fourth order differential operators [17, 18]. 

In this article, we choose three topics among them, and try to explain 
their flavours in Sections 2, 3 and 4, respectively with emphasis on the role of 
Sato's idea on algebraic analysis, both in philosophy and in techniques. For 
the reader's convenience, we list some representation theoretic properties of our 
minimal representation in the Appendix. 

§2. Conservative quantities for 2?-modules. 

The energy of a wave is a conservative quantity for the wave equation, 
namely, it is invariant under time-translations. In this section, we discuss higher 
symmetries coming from conformal transformations. By using the idea of Sato's 
hyperfunctions [20, 47], we construct conservative quantities for specific ultra- 
hyperbolic equations (see Theorem 2.6). 

§2.1. Yamabe operator and conformal geometry 

A diffeomorphism ft, of a Riemannian manifold (A, g) is said to be confor- 
mal if there exists a positive-valued function Sl(h, •) on X such that 

h*g hx = VL{h,x) 2 g x for x E X. 

It is isometry if il(h, •) = 1. We write 

Isom(A, g) C Conf (A, g) 

for the groups consisting of isometries and conformal diffeomorphisms, respec- 
tively. The same notation will be applied to a more general setting where 
g is a non-degenerate symmetric tensor, namely, to an indefinite-Riemannian 
manifold. 

The invariance for the Laplacian Ax characterizes isometries among diffeo- 
morphisms of A. In other words, a non-isometric transformation on (A, g) does 
not preserve Ax- However, the Laplacian Ax is still subject to the following 
covariance under conformal transformations: 

Wn+2 (h) o Ax = Ax o vDn^2 (h) for any h E Conf (A, g), (2.1) 

where n is the dimension of A, Scalx is the scalar curvature, and 

~ n — 2 

Ax :=Ax — 77 tt Scalx (the Yamabe operator), 

4(n — 1) 

w x (h)f(x) :=0(ft- 1 ,x) A /(ft- 1 x) for / E C°°(X). 
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The formula (2.1) implies that the operator Ax (or Ax) is not conformally 
invariant, but the P-module generated by Ax is conformally invariant ! As far 
as the solutions are concerned, the invariance of the I?- module is sufficient. 
Namely, by putting 

Sol(A x ) := {/ € C™{X) : A x f = Scal x /}, (2.2) 

we get: 

Fact 2.1. The conformal group Conf(X, g) preserves Soli Ay) via vj n -2 . 
See [38, Theorem 2.5] for the proof. 
Remark 2.2. 

1) Other eigenspaces Sol(A x — A) are not conformally invariant if A ^ 0. 

2) It may be better formulated if we use the ring of twisted differential oper- 
ators acting on sections of the line bundle £ n -2 . 

2 

3) The differential equation A x f = 0, namely, Axf — Scalx/, is ellip- 
tic, hyperbolic, or ultra-hyperbolic, respectively, if (X, g) is Riemannian, 
Lorentzian, or of general signature, respectively. 

Then a general problem is: 

Problem 2.3 (see [30, Problem C]). 

1) Does there exist an invariant inner product on an appropriate subspace of 
Sol(A x )? 

2) If yes, construct it explicitly. 

Such an inner product may be seen as a conservative quantity for the 
solution to the equation Axf = 0. Problem 2.3 does not find a final answer in 
the general setting. We shall give a partial answer in the flat case (see Theorem 
2.6 below). 

§2.2. Conservative quantities 

Let R p ' 9 be the Euclidean space M. p+q endowed with the flat indefinite- 
Ricmannian structure 

ds 2 = dx\ -\ h dx 2 - dx 2 p+l dx* +q . 
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Then, the corresponding Laplace-Beltrami operator takes the form: 



Q2 Q2 Q2 



If 



dx\ dxi dxl 



■p+1 



dxl 



p+q 



Obviously, the scalar curvature on W ,q vanishes identically. Hence, the Yamabc 
operator on W' q coincides with O p ^ q . The space of solutions to O p ^ q f = 0, 
denoted by Sol(D p _ q ), is obviously invariant under the motion group 



It was proved in [40, Theorem 4.7] that Sol(D p q ) has even larger symmetries 
if p + q is even, namely, by the indefinite orthogonal group 



G := 0(p + l,q+l) = {g€ GL(p + q + 2, K) : '.g/p+M+i.? = I P +i, q +i} 



acting on M. p+q as Mobius transforms. (To be more precise, G preserves the 
space SoIq{U\ pa ) of smooth solutions with certain decay conditions at infinity 
together with their derivatives.) 

Remark 2.4. 

1) The parity condition on p + q is crucial. In fact, a theorem of Howe and 
Vogan [51] asserts that there does not exist an infinite dimensional repre- 
sentation of G whose Gelfand-Kirillov dimension is p + q — 1 if p + q is odd 
and p, q > 3. 

2) Solo(0 Piq ) is defined as the twisted pull-back of smooth functions on the 
conformal compactification of W +q . See [40] for details. 

Problem 2.3 in this specific setting is stated as: 

Problem 2.5. Find a G -invariant inner product on Sol n (D Piq ) if exists. 



If p, q > and p + q is even and greater than two, then we can tell a priori 
that the representation on Solo(n\ p q ) is unitarizable and irreducible (e.g. [5, 
38]) by algebraic techniques. Namely, we know the existence and the uniqueness 
of a G-invariant inner product on Sol (O p ^ q ) in this case. 

What we seek for in Problem 2.5 is not merely an abstract unitarizabil- 
ity but the unitarization of the representation space for a concrete geometric 



Isom(R p '«) ~ 0(p,q) k W +q . 



§2.3. Unitarizability versus unitarization. 
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model, namely, the construction of the invariant inner product. Then, there 
are two approaches to the unitarization — one is easier, and the other is more 
challenging as discussed below. 

An easier approach to Problem 2.5 is to write the inner product by using 
the integral representation of solutions. Such an integral formula was given in 
[40, Theorem 4.7] by using an explicit formula of the Green kernel [30, 40]. The 
disadvantage of this approach is that the formula of the inner product involves 
a preimage of the integral representation, which is not canonically given. 

A second approach is to use an expansion of solutions into a countable 
sum of better understood solutions, and then to give a Parseval-Plancherel 
type theorem. We shall discuss this approach in Section 3. 

A more intrinsic approach is to find a formula of the inner product directly 
without using the integral representation of solutions. A hint to this is the well- 
known formula of energy to the wave equation, which is given by the integration 
of the Cauchy data on the hyperplane (t = constant) in the space-time, see 
(2.6). (We note that, however, the energy is not conformally invariant but 
invariant only under time-translations.) 

In order to explain the second approach, let us set up some notation. We 
recall that any non-characteristic hyperplane in W' g is written as 



for some c e R and v e M. p - q such that (v, v)^ P ,q = ±1. Fix such v, and express a 
function / on M. p+q as Sato's hyperfunction ([47]) in the direction of v, namely, 



Here, f±(x + tv) is a holomorphic function of one variable t near the real axis 
in ±Imi > 0. 



a = a VtC := {x e W +q : (x,v) MP , q = c] 



(2.3) 




(2.4) 



We set 



df± 




d_ 

Ft 



t=of(x + tv) 



(normal derivative) 



dv 



and introduce a function Q a f on the hyperplane a = a VyC by 




1 



df+ 



f- 



df- 
dv 



)• 



Finally, we define 




(2.5) 
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The right-hand side of (2.5) does not always converge, but it makes sense 
if / satisfies suitable decay conditions, say, / € Solo(O p . q ). Then, we can give 
an answer to Problem 2.5 as follows: 

Theorem 2.6 (see [40, Theorem 6.2], also [30]). 

1) For f € Solo(0 Pt q), (2.5) is independent of the choice of the pair (/+, /_) 
in the expression (2.4) and of the hyperplane a. 

2) (/, /) > for any f e Solo(D Ptq ). The equality holds if and only if f = 0. 

3) The polarization of the norm (2.5) yields a G-invariant inner product on 

Soi (n Ptg ). 

We denote by Solo(O p ^ q ) for the Hilbert space obtained as the completion 
of Solo(0 P:q ). Then, we get a unitary representation of G = 0(p + 1, q + 1), to 
be denoted by zu = w p+1 ' q+1 , on Solo(D Pyq ). It turns out that this is irreducible 
and a minimal representation of G. See Section 5 for representation theoretic 
properties of zu. 

Remark 2.7. The assertion 1) in Theorem 2.6 is a part of the invariance 
of the inner product ( , ) because any non-characteristic hyperplane is conjugate 
to either x\ = or x p+q = by the motion group Isom(IR p ' 9 ) ~ 0(p, q) K W +q . 
We note that G contains Isom(M M ) as a proper subgroup. 

The proof of Theorem 2.6 was given in [40] by using some representation 
theoretic results of the representation zu. It might be interesting to find a proof 
that does not depend on group theory but only on geometry such as Stokes' 
theorem. We then pin down this as an open problem: 

Problem 2.8. Give a purely geometric proof to Theorem 2.6. 



Our conformally invariant inner product (2.5) is very close to the energy 
of the wave, where one integrates Cauchy data on the zero time hyperplane. 
We end this section by making more explicit its connection. 

For p — 1, let us introduce time and space coordinates (t;x) instead of 
the previous coordinates (x\, ■ ■ ■ , x p ; x p+ \, ■ ■ ■ , x p+q ). Then, the energy of the 
wave / is given by 



§2.4. Energy generator 




(2.6) 
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Then, in terms of the inner product (2.5), £(/) is written as 

(f,\H\f) = (f + ,Hf + )-(f-,Hf-) 

where H = id t is the energy generator (infinitesimal time translations). Since 
the energy generator H is invariant under time-translations (i.e. invariant by a 
one-dimensional subgroup of G) and the inner product ( , ) is invariant under 
the whole group G, £(/) is also invariant under time-translations. This explains 
the classical fact that the energy (2.6) is a conservative quantity in the narrow 
sense that it is independent of which constant-time hyperplane we integrate 
over. 

§3. Qualitative branching laws 

In Section 1, we have given a concrete formula of the conformally invariant 
inner product on the minimal representation (conservative quantities). It is 
given by the integral over hyperplanes. Yet another formula of the same inner 
product will be given as a countable sum of well-understood quantities. 

This is a Parseval-type theorem (see Theorem 3.4), which is built on a 
'good expansion theorem' of solutions. Such an expansion theorem is obtained 
as a special case of the general theory of discretely decomposable restrictions of 
unitary representations (see Theorem 3.2). We will see that algebraic analysis 
provides a powerful method to branching problems in representation theory (cf. 
Problem 3.1 below). 

§3.1. Breaking symmetries and discrete decomposability 

Suppose 7r : G — > GL{H) is a unitary representation of a Lie group G. 
Given a subgroup G' of G, and consider the broken symmetry, namely, the re- 
striction ir\a' ■ In general, the restriction ir\ G > decomposes into a direct integral 
of irreducible representations of G' . Our concern here is with: 

Problem 3.1 (see [24, 25]). For which triple (G, G', n) does the restric- 
tion tt\g' decompose discretely with finite multiplicities? 

It often happens that the irreducible decomposition of the restriction tt\g' 
(branching law) contains continuous spectrum if G' is non-compact. Even 
worse, each irreducible representation of G' may occur in the branching law 
with infinite multiplicities. Thus, Problem 3.1 seeks for a very nice class of 
branching laws. 
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Now, let us fix some notation for a real reductive group G. Let K be 
a maximal compact subgroup of G, T a maximal torus of K, and t, £ the Lie 
algebras of T, K, respectively. We choose the set A + (t, t) of positive roots, and 
denote the dominant Weyl chamber by t+(C \/— It*). We also fix a K- invariant 
inner product on {, and regard ^/^lt* as a subset of ^J r ^lt* . 

Suppose that K' is a closed subgroup of K. The group K acts on the 
homogeneous space K/K' from the left, and then on the cotangent bundle 
T*(K/K') in a Hamiltonian fashion. We write 

fx : T*{K/K') -> V^lt* 

for the momentum map, and define the following closed cone by 

C K (K') := Image fi(~)t + . 

For a subgroup G' of G, we shall consider Ck(K') by setting K' := K n G' . 

Next, suppose that n is a (reducible) representation of a compact Lie group 
K. The asymptotic if -support of -k, to be denoted by ASk(k), was introduced 
by Kashiwara and Vergne [21] as the asymptotic cone of the K-types of n. 
From definition ASk(k) = {0} if dim7r < oo. For a representation n of G, we 
can define AS^M for the restriction 7r|^-. 

We are ready to state an answer to Problem 3.1: 

Theorem 3.2 (see [26]). Suppose that ir is a unitary representation of 
G of finite length, and that G' is a closed subgroup of G. We set K' = K n G' . 
If 

C K (K')nAS K (n) = {0}, (3.1) 

then the restriction tt\g> decomposes discretely into a direct sum of irreducible 
unitary representations of G' with finite multiplicities. 

An upper estimate of the singularity spectrum of the hyperfunction char- 
acter of 7r plays a crucial role in the proof of Theorem 3.2. In particular, the 
assumption (3.1) assures 

Restriction and Trace (hyperfunction character) commute. (3.2) 

Here, we remark that the character of an infinite dimensional representation n, 

Trace n(g) (g e G) 



does not make sense as an ordinary function because Trace 7r(e) = dim7r = oo. 
Harish-Chandra proved that Trace it is well-defined as a distribution on G if n 
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is an irreducible unitary representation of a real reductive group G, and proved 
further that Tracer belongs to L\ oc {G). On the other hand, the restriction 
Trace tt\k is not locally integrablc on K any more (see Atiyah [1]). What (3.2) 
means is that 

Trace(7r|x') = Trace(7r)|#-' 

as an identity of hyperfunctions (or distributions) on K' . See [26, Theorem 2.8] 
for the proof. We also refer to the lecture notes [32] for heuristic ideas of the 
proof. 

Recently, Hansen, Hilgert, and Keliny [13] has given an alternative proof 
of Theorem 3.2 by replacing Sato's hyperfunctions with Schwartz's distribu- 
tions. See also [27, 28] for the necessary condition of discrete decomposability 
of branching laws, where the associated variety of an infinite dimensional rep- 
resentation 7r (an analogue of the characteristic variety of a P-module) plays 
an important role. The references [29, 31] discuss some applications of discrete 
branching laws. 

Loosely, Theorem 3.2 says that if Ck(K') and AS^ (ir) are not 'large' then 
the restriction 7r| G / is discretely decomposable. We note that Ck(K') = {0} if 
K' = K, and consequently, the assumption (3.1) is automatically satisfied. In 
this case, Theorem 3.2 is nothing but Harish-Chandra's admissibility theorem 
([14]). For any minimal representation 7r of a reductive group G, we get from 
[51] that ASk (tt) is one-dimensional, i.e. AS^ (7r) = Rv or R + v where v is the 
highest root. Thus we can expect that there is a rich family of subgroups G' 
of G for which the restriction of the minimal representation of G decomposes 
discretely. 

§3.2. Space forms of indefinite Riemannian manifolds 

Before applying Theorem 3.2 to actual branching problems, we review 
quickly known results about the geometry and global analysis on space forms of 
indefinite-Ricmannian manifolds (referred also to as pseudo-hyperbolic spaces, 
generalized hyperboloids, etc.). 

We set 

X™ :={(x,y) e R p+1 ffiR* : ||x|| 2 - ||y|| 2 = 1} ~ 0(p + 1, q)/0(p, q), 
X p j q :={(x,y) G R p ®R q+1 : |M| 2 - |M| 2 = -1} ~ 0(p, q + l)/0(p, q). 

We note that A^'° ~ S p and X a j q ~ S q . By switching the factor, we have 
X p ' q ~ X q ' p 

We induce an indefinite Riemannian structure on X+ q and X^ q from the 
ambient space R p+1 ^ and RP>« +1 , respectively Then, X™ and X™ have 
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constant sectional curvatures. Here is a summary of indcfinitc-Ricmannian 
manifolds X™ and X p J q : 





sectional curvature k 


signature of metric tensor 


x p, q 


k = +1 


(p, l) 


x p_, q 


K = -1 


(P, l) 



Let L 2 (Xl' q ) be the Hilbert space of square intcgrable functions on 
X^T ' q with respect to the induced volume element. For A € C, we set 

Vt q := {/ e L\Xl-^) : A x p-i,«/ = (1 - A 2 )/}, 

where the Yamabe operator A^^ takes the following form: 

A xP+ , q = A X ™ - l -( P + q)(p + q-2). (3.3) 

Clearly, the isometry group Isom(X5~ ' q ) ~ 0(p,q) preserves V^' 9 for any 
A e C. The representations on V^' 9 are called discrete series representations 
for X?" 1 ' 9 if Vf 9 ^ {0}, which were studied by Gelfand, Graev, Vilenkin, 
Shintani, Molchanov, Faraut, and Strichartz among others. We summarise: 

Proposition 3.3. 

1) (p = 1) V p ' q = {0} for any A e C. 

2) (p ^ 1) V p ' q £ {0} A e 2^ + 2Z and A ^ 0. 

Furthermore, 0(p 7 q) acts irreducibly on each V^ q , when it is non-zero. 

The resulting representation in Proposition 3.3 2) will be denoted by tt x ' q . 
Since V p ' q = V p ' q , we may and do assume Re A > without loss of generality. 
By the coherent continuation of n^' q for A > such that A € p ^ + 2Z, we can 
define irreducible unitary representations 7Tq' 9 (p + g:even) and 7r^'? (p + q:odd) 

2 

of 0(p, g). These representations do not lie in L 2 (X^T ' 9 ) but enjoy analogous 
algebraic properties to 7r^' 9 (A > 0) (see [24, §6] or [39, §5.4] the vanishing 
results on cohomologies in details). 
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We return to the setting of Section 2.2. The flat indcfinitc-Ricmannian 
manifold W' q may be seen as the direct product of two flat spaces: 

(W, dx\ + --- + dx 2 p ) x (R«, -dx 2 p+1 dx 2 p+q ). 

Likewise, the direct product of two space forms: 

Y := X^' q ' x X p "< 9 " 

is locally conformal to MP' q for any p', q',p", q" such that 

p'+p"=p, q' + q" = q. 

This local conformal map is given as follows: For u — ((£o, ff), {£," , v"> Vo)) € 
R i+ P '+q' ® w"+i" +1 , we set 

*(«) ^^—(eW^'W')- 

so + % 

Then, the restriction of $ to Y is conformal (see [38, Lemma 3.3], for example). 
More precisely, the map 

$ : X^' q ' x X p l'- q " -> IR M (3.4) 

is well-defined and conformal in the open dense set Y' of Y, defined by £ +Vo 
0. Correspondingly, if we set 

(**/)(«) := (^— ) E± ^/($W) (3.5) 

then $*/ solves A Y $* f = on 1" if Dp,,/ = (sec [38, Proposition 2.6]). 
Here, Ay is the Yamabe operator on Y, which amounts to 

A Y =A x p',g' — A xp »,g» 

=A x ^, q , - A xP ,,, q „ - -^p 1 + q' -p" -q"){p + q-2). 

Hence, we can realize the minimal representation vo of 0(p + 1, q + 1) on the 
solution space AyF — as well through $*. 

We note that the map (3.4) is two to one at generic points. In order to give 
a global action of the group 0(p + 1, q + 1) on the solution space to AyF = 0, 
we need to define F = $*/ by (3.5) for £ + ?7o > 0, and by the parity condition 
for £ + Vo < so that F(-u) = (-l) £ ? J F(u) holds (sec [40, (4.4.2a)]). 
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In light that the isometry group of Y = ' q x X p _ ' q is the reductive 
group 0(p' + 1, q') x 0(p", q" + 1), it is natural to consider the branching law 
of the minimal representation zu with respect to the following symmetric pair 

0(p + l,q + l)i 0(p' + 1, q 1 ) x 0{p", q" + 1) 

by using the geometric model Y. 

In this setting, the criterion (3.1) of Theorem 3.2 holds if and only if p" = 
or q' = (see [39, Theorem 4.2]). Then, it follows from Theorem 3.2 that w 
decompose discretely. For the description of the irreducible decomposition, wc 
define the space of spherical harmonics of degree / by 

H l (R m ) :={ V G C 00 ^™- 1 ) : A Sm -icp 

= {^ec7 00 (5 m - 1 ):A 5m - 1 ^=|^-^+^) \<p\. (3.6) 

The orthogonal group 0(m) acts irreducibly on V. l (W n ) for any / <G N. 

Here is the branching law together with quantative information on the 
invariant inner product: 

Theorem 3.4 (see [39, Theorem B]). Suppose p + q (> 2) is even, q = 
(( + q" , and p,q > 0. Then the twisted pull-back $* of the conformal map 
$ : Y — > W' q induces the following quantative branching law: 

1) (branching law; 0(p + 1, q + 1) I 0(p + 1, q') x 0(q" + 1) ). 

OO ® 

^ P+M+1 |o( P+ i, 9 <)xO( 9 " + i) * E ® (3-7) 

i=o 2 2 

Here, the right-hand side of (3.7) is a multiplicity-free Hilbert direct sum 
of irreducible representations ofO(p+ l,q') x 0(q"). 

2) (Parseval-type theorem). For f G Solo(D p ^ q ), we expand <&*/ into i/ie 
series ^2 { Fi according to the discrete decomposition (3.7). Then we have: 



^=£G+y-*)ll^llV) (3-8) 



-ffere || ||rp,« is the norm defined in Theorem 2.6. 

In view of (3.6), the self-adjoint operator j — A Sq » is non-negative, and 

therefore we can define a pseudo-differential operator — ^s<j") 4 on y = 
A^' 9 ' x S*?" as well as on S"" . 
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Hence, we get another expression on the invariant inner product of the min- 
imal representation in the geometric model Y by means of a pseudo-differential 
operator: 

Corollary 3.1. Suppose p + q (> 2) is even, q = q' + q" , and p, q" > 0. 

Let — A s „//^ 4 be the pseudo-differential operator on Y = X^ q x S q " . We 
set F = $*/ for f G Sol (\3 p , q ). T/ien 

2 

(3.9) 

L 2 (Y) 

Remark 3.5. 

1) For q" = or 1, I + _ i < o if Z = 0. In this case V p+ ^; q '_ 1 = {0}. 

Nevertheless, we can justify the summand in (3.8) by using the argument 
of the analytic continuation. 

2) In the case p" = q' = 0, the branching law (3.7) is nothing but the K-type 
formula of the minimal representation w, and (3.8) was proved earlier by 
Kostant [42] for p = q = 3, and by Binegar and Zierau [5] for general p, q 
such that p + q is even and greater than 2. 

3) In the case q" — 0, we have Y ~ X+ q x S°, namely, Y consists of two 
copies of X^ q . Then Theorem 3.4 asserts that the minimal representation 
splits into two components, namely, 

p+l,g+l| ffi P+1,9 

w IO(p+l,g) — "_i ~i 

because ^(R 1 ) = for / > 2. 

4) In the case p" = and p' = q' = 1, we are dealing with the branching law 
for the pair 

0(2,9 + l)|0(2,l) xO(q). 

The branching law (3.7) in this special case yields a setting of the defor- 
mation of the Fourier transform (see Section 4) . 

§4. Deformation of Fourier transforms 

Minimal representations give us also a hint to define a generalization of 
the Fourier transform. In this section, we introduce a holomorphic semigroup 
Zk,a( z ) consisting of Hilbert-Schmidt operators with three parameters: 
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a: interpolating minimal representations of simple groups of type C and D, 
k: Dunkl deformation parameter (multiplicities on the root system), 
z: complex number, 
such that the operator-valued boundary value 

lim I k ,a( z ) 

Rezj,0 

of Hilbert-Schmidt operators yields a one-parameter group of unitary opera- 
tors. The underlying idea may be seen as a descendant of Sato's hyperfunc- 
tion theory [47] and also that of the Gelfand-Gindikin program [11, 44, 48] 
for unitary representations of real reductive groups. We shall see in Dia- 
gram 4.2 that the Euclidean Fourier transform, the Hankel-type transform, 
and the Dunkl transform, etc. arise naturally as the special values of Ik, a {^) = 

lim^oXfe,a(T+ £ )- 

§4.1. i 2 -model of minimal representations 

We return to the setting of Section 2.2. If a tempered distribution / e 
S'(IR P+9 ) satisfies the differential equation PjQ f — 0, then it is easy to see that 
its Fourier transform J 7 / is supported on the characteristic variety 

S := {£ e M. p+q + = 0}. (4.1) 

Much more than this, the following theorem holds (see [39, Theorem 6.2]): 

Theorem 4.1. For p + q > 2 even and p,q > 0, the Euclidean Fourier 
transform T = Jrp+s induces the bijection: 

T : Sol {D Piq ) ^ L 2 (E). 

It is an isometry up to the scalar multiplication by 2" + 2 +2 -k f+ 2 +1 . 

Here, Solo(O p , q ) is the Hilbcrt space with respect to the conservative quan- 
tity ( , ) defined in Theorem 2.6, and L 2 (S) denotes the Hilbert space consisting 
of square integrable functions with respect to the canonical measure on H. The 
non-trivial part of Theorem 4.1 is to show that Image J 7 n £ 2 (S) ^ {0}. See 
[39, Theorem 6.2] for the proof. 

It follows from Theorem 4.1 that we can realize the minimal representation 
of the indefinite orthogonal group 0(p + 1, q + 1) on the Hilbert space L 2 (E) 
(L 2 -model) from the one on Sol (C\ p ^ q ) (conformal model). 
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At this moment, we remark a distinguishing feature of minimal representa- 
tions (see Appendix in Section 5). Unlike well-understood family of irreducible 
unitary representations of real reductive groups such as unitary principal series 
representations or discrete series representations, minimal representations are 
too 'small' that there is no existing geometric model for which both group ac- 
tions and the Hilbert structure are given in a simple manner (cf. [6, 50]). We 
pin down the advantages of the aforementioned two models: 





Group action 


Hilbert structure 


Conformal model Sol(D Ptq ) 


simple 


® 


L 2 -modcl L 2 (E) 


® 


simple 



Finding the missing parts (T) and @ is interesting, particularly because it 
interacts with other branches of mathematics. Representation theoretic con- 
sideration plays a guiding role in formalizing problems there. In fact, we have 
seen in Theorem 2.6 that (T) brought us to the construction of conservative 
quantities for ultra-hyperbolic equations, whereas (2) leads us to the notion of 
a Fourier transform on the isotropic cone S [3, 35, 36], as discussed below. 

From now, we consider the missing part (2). In order to find the global 
formula of group actions on the i 2 -model, let us clarify what is trivial and 
what will be the crucial operator. We observe that there is a maximal parabolic 
subgroup P of G = 0(p + 1, q + 1) that contains the conformal transformation 
group 

Conf(IR M ) ~ (M >0 x 0(p,q)) x R p+q 
as a subgroup of index two. Then we have the Bruhat decomposition 

G = P II PwP, 

where w = ( ^ p+1 ^ | . In fact, the Euclidean Fourier transform Jdn ap- 

V o -i g +i) 

pear as the unitary inversion operator of the Segal-Shale- Weil representation 
of the metaplectic group Mp(N,M), which is also a minimal representation. 
See [37, Chapter 1] for the comparison of T~, and TgN from this point of view. 

In the L 2 -model of the minimal representation 7r of G on L 2 (E), the in- 
action is simple, namely, it is given just by translations and multiplications 
[40]. Hence, it is enough to find the single unitary operator (unitary inversion 
operator) n(w) in order to fill the missing part (2). We set 



J" s := cn(w), 



(4.2) 
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where c is the phase factor. Algebraically, Ts. intertwines the multiplication of 
coordinate functions £j (1 < j < p + q) with the Bargmann-Todorov operators 
Rj (1 < j < P + q) which are mutually commuting differential operators of 
second order on S (see [2], [36, Chapter 1]). 

This algebraic feature is similar to the classical fact that the Euclidean 
Fourier transform intertwines the multiplication operators £j and the dif- 
ferential operators \f^-idj (1 < j < N). 

The goal of this section is to explain these operators T-s and J" r n from 
a broader point of view, by constructing continuous family of operators that 
include T-s and J* k at as their special values. 

For this, we limit ourselves to the case p = 1. Then, the light cone S (see 
(4.1)) splits into the forward light cone S + and the backward light cone S_ 
according as x\ > and x\ < 0. The unitary inversion operator T~. preserves 
the direct sum 



For later purpose, we set q = N. Then the projection to the second factor, 
R 1 ©R^ — > R N , induces the following isomorphism between the Hilbert spaces: 



Via (4.4), the unitary inversion operator on L 2 (E + ) may be seen as a unitary 
operator on L 2 (M. N , ||x|| ~ 1 dx). The explicit formula of J 7 ^ in the coordinates 
of R w was given in [35]. In this framework, we can construct a holomorphic 
family of bounded operators so that the unitary operator T-= is obtained as the 
limit of holomorphic objects. Deformation in the Dunkl setting [3, 4] is also 
built on this formulation. We will discuss those operators in this generality in 
Section 4.4. 

An alternative approach was taken in [36] based on the Barnes-Mellin 
integral to find the kernel function of T~. for general p and q. 

§4.2. Hermite semigroup and Fourier transform 

We begin with recalling a general fact on the classical Hermite operator 



L 2 (S) = L 2 (S + )®L 2 (S_). 



(4.3) 



L 2 (E + ) ~ L^MMr 1 ^). 



(4.4) 



on R 



(e.g. [9, 19]): 




(4.5) 
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Then, A — ||a;|| 2 extends to a sclf-adjoint operator on L 2 (R N ). We normalize 
the Euclidean Fourier transform J" r jv on L 2 (R N ) as 

= -^r I f{x)e-^dx. 

Then, J" r n is written as a special value of the one-parameter group of unitary 
operators 

X (t) :=exp(^(A-|M| 2 ) 



x 2 

namely, we have 

Jk-=e^ Jv exp( ^(A-Hsf i ) (4.0) 



^ 4 

Further, the one-parameter group x(i) of unitary operators extends to a holo- 
morphic semigroup I(z) defined by 

I(z) = cxp^(A- ||x|| 2 ) forRcz>0. (4.7) 

The semigroup I(z) is called the Hermite semigroup, and it is expressed as 
an integral transform against the Mehler kernel [9, 19], a Gaussian type kernel. 
Next, we consider another differential operator on R N , 

\\x\\A — \\x\\. 

It turns out that this operator has a self-adjoint extension on the Hilbert space 
L 2 (R N , \\x\\~ 1 dx) (see [35, Section 1.1]). Moreover, an analogous formula to 
(4.6) holds: via the identification (4.4), the 'Fourier transform' on the 
forward light cone S + can be expressed as 

J- H = C exp(^(||x||A-|MI)), (4-8) 

where c = e5 1 "( JV_1 ) is the phase factor. Then, the expression (4.8) allows 
us to see T~. as the limit of the following holomorphic semigroup (Laguerre 
semigroup) 

l(z)=expz(\\x\\A-\\x\\), forRez>0, (4.9) 

as z ^ + 0. The kernel function of l(z) is given in terms of the Bcssel 
function [33]. 

Interpolating A— ||a:|| 2 and ||a;|| A— ||a;||, namely, the infinitesimal generators 
of the Hermite semigroup (4.7) and the Laguerre semigroup (4.9), we consider 
the differential operator 

A , a :=|M| 2 - a A-Nr- 
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It might not be so obvious that the symmetric operator Ao, a has a self-adjoint 



extension on the Hilbcrt space L 2 



\x\\ a 2 dx). In fact, it is the case. The 



proof uses representation theory (see Proposition 4.6), and the same idea works 
in a more general setting of Dunkl's differential-difference operators. Thus, we 
shall introduce a holomorphic semigroup Ik,a(z) with infinitesimal generator 
^k,a (see (4.10) below for the definition) for Rez > with parameters k and a 
in Section 4.3. 

In Diagram 4.2, we have summarised some of the deformation properties 
by indicating the limit behaviour of the holomorphic semigroup X fciQ (z). The 
specialization Tk,a{^) leads us to a (k, a)- generalized Fourier transform T^.a 
(up to a phase factor), which reduces to the Fourier transform (a = 2 and 
k = 0), the Dunkl transform T>k (a — 2 and k = 0), and the Hankel-type 
transform (a — 1 and k = 0). 




(k, a)-generalized Fourier transform 








holomorphic semigroup Xk,a(z) 



a-s-2 



T-k,i{z) 



Dunkl transform 
T\ 



Hcrmitc semigroup 
I(z) 



a— yl 



Fourier transform 



'unitary inversion operator 




the Weil representation of 
Mp(N, R) 



the minimal representation of 
0(2,JV + 1) 



Diagram 4.2. Special values of holomorphic semigroup Xk,a(z) 



§4.3. Holomorphic semigroup X kM (z) with two parameters k and a 
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This subsection introduces a holomorphic semigroup, denoted by 2k.a{z), 
of which the infinitesimal generator is a self-adjoint differential-difference op- 
erator. 

Let € be the Coxeter group associated with a reduced root system 7Z in 
R N . For a ^-invariant function k = (k a ) (multiplicity function) on TZ, we set 

:= 2 

and write A& for the Dunkl Laplacian on (see [16]). This is a differential- 
difference operator, which reduces to the Euclidean Laplacian A when k = 0. 
We take a > to be yet another deformation parameter, and define 

A fc> „ := Hzf-Afc - H . (4.10) 

We define a density on R N by 

k , a (x) :=\\x\\ a - 2 [] \(<x,x)\ ka . 

The volume of the unit ball with respect to the measure "&k,a(x)dx is explicitly 
known in terms of the gamma function owing to the work by Selberg, Mac- 
donald, Hcckman, and Opdam among others but we do not go into details (see 
Etingov [8]). 

In the case a = 2 and k = 0, $o,2( x ) = 1 an d A 0i 2 is the Hermite operator 
(4.5) on R N . 

Here are basic properties of our differential-difference operator Ak, a '- 

Theorem 4.2 (see [4, Theorem A]). Assume a > and a + 2(k) + N — 
2 > 0. 

1) Afe i0 extends to a self-adjoint operator on the Hilbert space L 2 (R N , ■dk^ a {x)dx). 

2) There is no continuous spectrum of Afc )0 . 

3) All the discrete spectrum of Ak, a is negative. 

We introduce the following operators on L 2 (R N , $k,a(x)dx) by 

lk, a {z) := exp(^A M ), for Rez > 0. (4.11) 

Correspondingly to the properties of the infinitesimal generator ^A fe>0 in The- 
orem 4.2, we get: 
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Theorem 4.3 (see [4, Theorem B]). Retain the assumption of Theorem 

12. 

1) {Tk, a (z) : Rez > 0} forms a holomorphic semigroup in the complex right- 
half plane {z G C : Rez > 0} in the sense that lk,a{z) is a Hilbert-Schmidt 
operator on L 2 (R N ,-& kt0/ (x)dx) satisfying 

lk,a( Z l) oI k,a{ Z 2) = Zk,a( Z l + (Re Zi , Re Z 2 > 0), 

and that the scalar product (T k . a {z)f,g) is a holomorphic function of z for 
Rez > 0, for any f,g G L 2 (R N , d k , a {x)dx). 

2) Zk.a(z) is a one-parameter group of unitary operators on the imaginary axis 
Rez = 0. 

We shall call T ka fz) as the (fc, a) -generalized Laguerre semigroup T ka {z). 
We note that Iq^{z) is the Hcrmite semigroup (4.7) (see [9, 19]), and 2^o,i( z ) 
is the Laguerre semigroup (4.9) (see [33]). 



§4.4. (fc, a)-generalized Fourier transforms T k . a 

Theorem 4.3 2) asserts that the 'boundary value' of the holomorphic semi- 
group Xk,a(z) produces a one-parameter family of unitary operators. 

The case z = gives the identity operator, namely, Xfc,a(0) = id. The 
particularly interesting case is when z — ^. We set 

c := cxp(z7r ^ T_jijjj j_ a — (phase factor), 

and define the (fc, a) -generalized Fourier transform by 

-Ffc,a ■= Cl fc,a(y) = CCX P (£(||x|| 2 - a A fe - ||x|| a )). 

Then, this operator T k ^ a for general a and k satisfies the following significant 
properties: 

Theorem 4.4 ([4, Theorem D]). Retain the assumption of Theorem 4-2- 

1) Tk,a "is a unitary operator on L 2 (R N , ■d k ^ a {x)dx). 

2) -^fe.a ° H ka = —H ka o T kM . Here, H ka is the differential operator of first 
order defined in (4.12). 
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3) T Ka °\\x\\ l 
Fk, a o (||z| 



a 




_a Afc o FkM, 
-\\A a °Fk, 



4) ^fe.a is of finite order if and only if a € Q. lis order is 2m if a = ™ swc/i 
i/iai (m, n) = 1. In particular, T k .\ is of order 2, and J 7 /^ «s o/ order 4. 

As indicated in Diagram 4.2, Fk, a reduces to the Euclidean Fourier trans- 
form F on R N if k = and a = 2; to the Dunkl transform 2?^ introduced by 
C. Dunkl himself if k > and a = 2. The unitary operator J"o ; i arises as the 
unitary inversion operator T~. on L 2 (S + ) of the minimal representation of the 
conformal group (see Section 4.1). 

Our study also contributes to the theory of special functions, in particular 
orthogonal polynomials; indeed we derive several new identities, for example, 
the (k, a)-dcformation of the classical Bochncr-Hcckc identity where the Gaus- 
sian function and harmonic polynomials in the classical setting (k = and 
a = 2) are replaced respectively with exp(— - \\x\\ a ) and polynomials annihi- 
lated by the Dunkl Laplacian. The (k, a) -generalized Fourier transform T k ,a 
also satisfies a Hcisenberg-type inequality. This generalizes the classical case 
(k = and a = 2) and Roslcr's Hcisenberg inequality [45] (k > and a = 2). 
We refer to [4] for full details. 

§4.5. Hidden symmetries in the Hilbert space L 2 (R N , dk,a{x)dx) 

The key idea of the proof for Theorem 4.1, 4.2, and 4.3 is to use more 
operators rather than the single operator A/j ja , and then to appeal representa- 
tion theory of s^, in particular, the theory of discretely decomposable unitary 
representations. 

Lemma 4.5. Let k be a multiplicity-function on a root system, and 
a e C x . Then, the following differential-difference operators on R N \ {0} 




\\x\\ 



a 



a 




x\\ 2 - a A k , 




(4.12) 



form an sl 2 -triple, namely, we have: 



[Hk, a , E^J - 2E+ a , [H k , a , E k J 



2E k,a> l E k,a> E k,a\ = H k,a- 
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Special cases of Lemma 4.5 was previously known: the case k = and 
a = 2 is the classical harmonic s^-triple (e.g. Howe [19]), the case k > and 
a = 2 by Heckman [16], and k = and a — 1 by Kobayashi and Mano [33]. 
The operator Ak, a (see (4.10)) takes the form, 

A M = ai(E+ a - E^ a ), 

which may be seen as an element of sl(2, C). 

Lemma 4.5 fits well into the framework of discretely decomposable repre- 
sentations of reductive groups [25, 26, 27] as we discussed in Section 3.1: 

Proposition 4.6 (see [4, Theorem 3.31]). If a > and a + 2{k) + N - 
2 > 7 then the representation of sl(2,R) lifts to a unitary representation of 
the simply- connected group on L 2 (R N , dk, a {x)dx). The resulting unitary rep- 
resentation is discretely decomposable, and commutes the obvious action of the 
Coxeter group £. 

This unitary representation plays the central role in the key to the proof of 
Theorems 4.2, 4.3 and 4.4. An explicit formula of the irreducible decomposition 
of L 2 (R N , $k,a(x)) is found in [4, Theorem 3.28]. In the special cases fc = and 
a, = 1 or 2, this formula may be regarded as the branching law of the minimal 
representations of 0(2, N + 1) or Mp(N, R), respectively (see Diagram 4.5 
below). Correspondingly, all the spectrum of A^ ia is also obtained explicitly. 

In the case a = 2, the sLj-action also induces the representation of SL(2, C) 
on the algebra generated by Dunkl's operators, multiplication operators, and 
the Coxeter group. The restriction of this action to SL(2,'E) coincides with a 
special case of the SL(2, Z)-action discovered by Cherednik [7] on the (degen- 
erate) rational DAHA (double afhne Hecke algebra). 

Theorem 4.6 asserts that the Hilbert space L 2 (R N , i9fc^ a (x)dx) has a sym- 
metry of the direct product group £ x SL(2, R) for all k and a. This symmetry 
becomes larger for special values of k and a as below: 



£x SL{2,\ 



fc^O 



> 0(N) x SL(2, 



(fc, a: general) 



a -> 1 



a -> 2 



0(2,A + 1) 



Mp(N, R) 



Diagram 4.5. Hidden symmetries in L 2 (R N , r dk,a{x)dx) 



For a = 2, this symmetry is given by the Segal-Shale-Weil representation 
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of the metaplectic group Mp(N, R). For a = 1, it is given by the irreducible 
unitary representation of the double covering (3(2, N + 1) of the conformal 
group on L 2 (M. N , ||a;|| _1 (fo;). Here, as we saw in Theorem 4.1, we do not need 
to take a double cover when N is odd. Both of them arc minimal represen- 
tations and, in particular, they attain the minimum of their Gclfand-Kirillov 
dimensions among the unitary dual. In this sense, our continuous parameter 
a > interpolates the i 2 -modcls of two minimal representations of different 
reductive groups by keeping smaller symmetries O(N) x SX(2,R). In view of 
Lemma 4.5, the (fc, a)-generalized Fourier transform Fk, a (k = 0, a = 1, 2) arise 
as the unitary operators (up to phase factors) corresponding to the following 
element, 



§5. Appendix — representation theoretic properties of w 

For the reader's convenience, we list some representation theoretic prop- 
erties of the irreducible unitary representation w of the indefinite orthogonal 
group G = 0(p + l,q + l), on which geometric analysis is the motif throughout 
this article. 

In what follows, we assume 



We write K ~ 0(p + 1) x 0(q + 1) for a maximal compact group of G, g ~ 
o(p+l, q+1) for the Lie algebra of G, Qc — °(P+9+2, C) for its complexification, 
and Qc = tc + Pc for the complexified Cartan decomposition. 

Here are some properties of w from representation theoretic viewpoints. 

1) vj is an irreducible unitary representation of G. 

2) (minimal representation) The representation w is a minimal representa- 
tion in the sense that the annihilator of the underlying (rjc, if )-modulc wk 
in the universal enveloping algebra U(gc) is the Joseph ideal if p + q > 6 
([5, 42]). See [10] for algebraic aspects of minimal representations of reduc- 
tive groups and the definition of the Joseph ideal. 

3) (restriction to the identity component) The group G has four connected 
components, and we write G = SO a (p + l,q + 1) for the identity com- 
ponent. Then, vj stays irreducible when restricted to G if and only if 
P,Q > 1- 



ex P 77 




p, q > 1 and p + q is an even integer greater than two. 
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4) (highest weight module case) If p = 1 or q = 1, then the restriction vj\g 
is a direct sum of two irreducible representations, one is a highest weight 
representation w+ and the other is a lowest weight representation nj_ . As 
we have seen in (4.3), this decomposition vj\g = kj + ® n7_ corresponds to 
the direct sum 

L 2 (~)=L 2 (~ + )®L 2 (S_) 

in the L 2 -modcl. Both w + and w- arc minimal representations of the 
connected group Go- 

To fix the notation, we suppose p = 1. Then, G is the conformal group 
0(2, q + 1) of the Minkowski space R 1,9 , namely, the Euclidean space R 1+<? 
equipped with the flat Lorentz metric of signature (l,q). In this case 
our representation vj has a long history of study, also in physics (see e.g. 
Todorov [49]). The minimal representation w + may be interpreted as the 
symmetry of the solution space to the mass-zero spin-zero wave equation. 
The representation zu + arises also on the Hilbcrt space of bound states of 
the Hydrogen atom. 

5) (spherical case) The underlying (gc, iQ-module vjk has the following K- 
type formula: 

m K ~ H a (R p+1 ) mn b (R q+1 ). (5.1) 

a,beft. 

In particular, the representation w is spherical (i.e. contains a non-zero 
if -fixed vector) if and only if p = q. 

6) (infinitesimal character) Let 3(flc) be the center of U(qc)- Then, the in- 
finitesimal character of wk is given by 

(1,^-1,^-2,.. ,1,0). 

Here, we normalize the Harish-Chandra isomorphism for the simple Lie 
algebra g c of type D n (n = ^ + 1), 

Hom c _ algcbra (3(flc),C) ~ C n /W(D n ), 

in a way that the infinitesimal character of the trivial one dimensional 
representation is 

(^,^-1,^-2,-, 1,0). 
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7) (theta correspondence) The representation w is obtained also as the theta 
correspondence of the trivial one-dimensional representation of SL(2,R) 
with respect to the following reductive dual pair 

0(p + SL(2, R) C Sp(p + q + 2,R). 

See [52]. 

8) (Gelfand-Kirillov dimension) The Gelfand-Kirillov dimension of the rep- 
resentation w, to be denoted by DIM(tn), attains its minimum among all 
unitary representations of G, that is, 

DIM(tu) =p + q-l. 

The associated variety of the underlying (qc, iv~)-module wk is given by 

AV( ro ) = ^ in n Pc , 

see [39, Lemma 4.4]. Here, C^ in is the minimal nilpotent coadjoint orbit 
in identified with the Lie algebra gc- 
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